Reference Documentation Refactoring by mgyucht · Pull Request #467 · databricks/databricks-sdk-py

mgyucht · 2023-12-01T13:01:23Z

Changes

This PR makes a number of changes to the documentation with the aim of improving the coverage of reference material as well as making the reference documentation more intuitive to follow by mirroring the structure of the library.

The biggest change here is the architecture: we're no longer generating ReStrucutredText by hand, but rather using the automodule directive to generate most of the documentation for us. This means that all classes and enums automatically have documentation, type annotations are present for all parameters, and links and references work properly. The main downside is that examples are no longer included in the main documentation, so I've added a page to the top-level TOC to point out the examples in our repo directly.

To improve the documentation coverage of existing classes, I tweaked some of our templates to include docs from the OpenAPI spec directly in the WorkspaceClient/AccountClient definitions, as well as in the generated code for dataclasses and enums. There are many things that are poorly documented now; we'll need to lean on better linting in the upstream OpenAPI spec to prevent this from getting worse.

Tests

make test run locally
make fmt applied
relevant integration tests applied

mgyucht · 2023-12-01T13:50:07Z

.codegen.json

      "pip install '.[dev]'"
    ],
    "post_generate": [
-      "pytest -m 'not integration' --cov=databricks --cov-report html tests",


add this back in

mgyucht · 2023-12-01T13:58:36Z

docs/gen-client-docs.py

+''')

-  {all}''')
+    def write_client_reference(self, title: str, class_name: str, filename: str):


mgyucht · 2023-12-01T13:58:43Z

docs/gen-client-docs.py

-        return "\n".join(out)
-
-
 @dataclass


pietern

Really nice to have inline docs for the methods and data classes.

## Changes #467 changed the docs to require the databricks SDK itself to be installed for Sphinx to generate docs correctly. This PR fixes the requirements file used by readthedocs.org to install the SDK before trying to generate docs. The requirements file is consumed from the root directory by readthedocs.py, so I moved it to the root directory so it's more obvious that it ought to be run from there. ## Tests  - [ ] `make test` run locally - [ ] `make fmt` applied - [ ] relevant integration tests applied

Bugfixes: * Fixed accidental rename ([#471](#471)). * Fixed parsing of ISO date strings ([#473](#473)). Other changes: * Updated GCP OAuth Readme ([#464](#464)). * Reference Documentation Refactoring ([#467](#467)). * Installed local library when generating docs ([#469](#469)). * Fixed readme links in pypi ([#472](#472)). * Updated a note for installing Python SDK on Databricks Runtime 13.1+ ([#474](#474)). * Updated GCP auth readme ([#470](#470)). API Changes: * Changed `update()` method for [w.connections](https://databricks-sdk-py.readthedocs.io/en/latest/workspace/connections.html) workspace-level service with new required argument order. * Added `cloudflare_api_token` field for `databricks.sdk.service.catalog.CreateStorageCredential`. * Added `cloudflare_api_token` field for `databricks.sdk.service.catalog.StorageCredentialInfo`. * Changed `name` field for `databricks.sdk.service.catalog.UpdateCatalog` to be required. * Added `new_name` field for `databricks.sdk.service.catalog.UpdateCatalog`. * Changed `name` field for `databricks.sdk.service.catalog.UpdateConnection` to no longer be required. * Added `new_name` field for `databricks.sdk.service.catalog.UpdateConnection`. * Changed `name` field for `databricks.sdk.service.catalog.UpdateExternalLocation` to be required. * Added `new_name` field for `databricks.sdk.service.catalog.UpdateExternalLocation`. * Added `new_name` field for `databricks.sdk.service.catalog.UpdateMetastore`. * Added `new_name` field for `databricks.sdk.service.catalog.UpdateRegisteredModelRequest`. * Added `new_name` field for `databricks.sdk.service.catalog.UpdateSchema`. * Changed `name` field for `databricks.sdk.service.catalog.UpdateStorageCredential` to be required. * Added `cloudflare_api_token` field for `databricks.sdk.service.catalog.UpdateStorageCredential`. * Added `new_name` field for `databricks.sdk.service.catalog.UpdateStorageCredential`. * Added `new_name` field for `databricks.sdk.service.catalog.UpdateVolumeRequestContent`. * Added `cloudflare_api_token` field for `databricks.sdk.service.catalog.ValidateStorageCredential`. * Added `databricks.sdk.service.catalog.CloudflareApiToken` dataclass. * Removed `continuous` field for `databricks.sdk.service.jobs.BaseRun`. * Removed `continuous` field for `databricks.sdk.service.jobs.Run`. * Changed `job_parameters` field for `databricks.sdk.service.jobs.RunJobTask` to `databricks.sdk.service.jobs.ParamPairs` dataclass. * Added `run_if` field for `databricks.sdk.service.jobs.SubmitTask`. * Added `run_job_task` field for `databricks.sdk.service.jobs.SubmitTask`. * Changed `update_config()` method for [w.serving_endpoints](https://databricks-sdk-py.readthedocs.io/en/latest/workspace/serving_endpoints.html) workspace-level service with new required argument order. * Added `put()` method for [w.serving_endpoints](https://databricks-sdk-py.readthedocs.io/en/latest/workspace/serving_endpoints.html) workspace-level service. * Added `rate_limits` field for `databricks.sdk.service.serving.CreateServingEndpoint`. * Changed `served_models` field for `databricks.sdk.service.serving.EndpointCoreConfigInput` to no longer be required. * Added `auto_capture_config` field for `databricks.sdk.service.serving.EndpointCoreConfigInput`. * Added `served_entities` field for `databricks.sdk.service.serving.EndpointCoreConfigInput`. * Added `auto_capture_config` field for `databricks.sdk.service.serving.EndpointCoreConfigOutput`. * Added `served_entities` field for `databricks.sdk.service.serving.EndpointCoreConfigOutput`. * Added `served_entities` field for `databricks.sdk.service.serving.EndpointCoreConfigSummary`. * Added `served_entities` field for `databricks.sdk.service.serving.EndpointPendingConfig`. * Added `extra_params` field for `databricks.sdk.service.serving.QueryEndpointInput`. * Added `input` field for `databricks.sdk.service.serving.QueryEndpointInput`. * Added `max_tokens` field for `databricks.sdk.service.serving.QueryEndpointInput`. * Added `messages` field for `databricks.sdk.service.serving.QueryEndpointInput`. * Added `n` field for `databricks.sdk.service.serving.QueryEndpointInput`. * Added `prompt` field for `databricks.sdk.service.serving.QueryEndpointInput`. * Added `stop` field for `databricks.sdk.service.serving.QueryEndpointInput`. * Added `stream` field for `databricks.sdk.service.serving.QueryEndpointInput`. * Added `temperature` field for `databricks.sdk.service.serving.QueryEndpointInput`. * Changed `predictions` field for `databricks.sdk.service.serving.QueryEndpointResponse` to no longer be required. * Added `choices` field for `databricks.sdk.service.serving.QueryEndpointResponse`. * Added `created` field for `databricks.sdk.service.serving.QueryEndpointResponse`. * Added `data` field for `databricks.sdk.service.serving.QueryEndpointResponse`. * Added `id` field for `databricks.sdk.service.serving.QueryEndpointResponse`. * Added `model` field for `databricks.sdk.service.serving.QueryEndpointResponse`. * Added `object` field for `databricks.sdk.service.serving.QueryEndpointResponse`. * Added `usage` field for `databricks.sdk.service.serving.QueryEndpointResponse`. * Changed `workload_size` field for `databricks.sdk.service.serving.ServedModelInput` to `databricks.sdk.service.serving.ServedModelInputWorkloadSize` dataclass. * Changed `workload_type` field for `databricks.sdk.service.serving.ServedModelInput` to `databricks.sdk.service.serving.ServedModelInputWorkloadType` dataclass. * Added `task` field for `databricks.sdk.service.serving.ServingEndpoint`. * Added `task` field for `databricks.sdk.service.serving.ServingEndpointDetailed`. * Added `databricks.sdk.service.serving.Ai21LabsConfig` dataclass. * Added `databricks.sdk.service.serving.AnthropicConfig` dataclass. * Added `databricks.sdk.service.serving.AutoCaptureConfigInput` dataclass. * Added `databricks.sdk.service.serving.AutoCaptureConfigOutput` dataclass. * Added `databricks.sdk.service.serving.AutoCaptureState` dataclass. * Added `databricks.sdk.service.serving.AwsBedrockConfig` dataclass. * Added `databricks.sdk.service.serving.AwsBedrockConfigBedrockProvider` dataclass. * Added `databricks.sdk.service.serving.ChatMessage` dataclass. * Added `databricks.sdk.service.serving.ChatMessageRole` dataclass. * Added `databricks.sdk.service.serving.CohereConfig` dataclass. * Added `databricks.sdk.service.serving.DatabricksModelServingConfig` dataclass. * Added `databricks.sdk.service.serving.EmbeddingsV1ResponseEmbeddingElement` dataclass. * Added `databricks.sdk.service.serving.EmbeddingsV1ResponseEmbeddingElementObject` dataclass. * Added `databricks.sdk.service.serving.ExternalModel` dataclass. * Added `databricks.sdk.service.serving.ExternalModelConfig` dataclass. * Added `databricks.sdk.service.serving.ExternalModelProvider` dataclass. * Added `databricks.sdk.service.serving.ExternalModelUsageElement` dataclass. * Added `databricks.sdk.service.serving.FoundationModel` dataclass. * Added `databricks.sdk.service.serving.OpenAiConfig` dataclass. * Added `databricks.sdk.service.serving.PaLmConfig` dataclass. * Added `databricks.sdk.service.serving.PayloadTable` dataclass. * Added `databricks.sdk.service.serving.PutRequest` dataclass. * Added `databricks.sdk.service.serving.PutResponse` dataclass. * Added `databricks.sdk.service.serving.QueryEndpointResponseObject` dataclass. * Added `databricks.sdk.service.serving.RateLimit` dataclass. * Added `databricks.sdk.service.serving.RateLimitKey` dataclass. * Added `databricks.sdk.service.serving.RateLimitRenewalPeriod` dataclass. * Added `databricks.sdk.service.serving.ServedEntityInput` dataclass. * Added `databricks.sdk.service.serving.ServedEntityOutput` dataclass. * Added `databricks.sdk.service.serving.ServedEntitySpec` dataclass. * Added `databricks.sdk.service.serving.ServedModelInputWorkloadSize` dataclass. * Added `databricks.sdk.service.serving.ServedModelInputWorkloadType` dataclass. * Added `databricks.sdk.service.serving.V1ResponseChoiceElement` dataclass. * Removed [a.account_network_policy](https://databricks-sdk-py.readthedocs.io/en/latest/account/account_network_policy.html) account-level service. * Removed `databricks.sdk.service.settings.AccountNetworkPolicyMessage` dataclass. * Removed `databricks.sdk.service.settings.DeleteAccountNetworkPolicyRequest` dataclass. * Removed `databricks.sdk.service.settings.DeleteAccountNetworkPolicyResponse` dataclass. * Removed `databricks.sdk.service.settings.ReadAccountNetworkPolicyRequest` dataclass. * Removed `databricks.sdk.service.settings.UpdateAccountNetworkPolicyRequest` dataclass. * Removed `name` field for `databricks.sdk.service.sharing.UpdateCleanRoom`. * Changed `name` field for `databricks.sdk.service.sharing.UpdateProvider` to be required. * Added `new_name` field for `databricks.sdk.service.sharing.UpdateProvider`. * Changed `name` field for `databricks.sdk.service.sharing.UpdateRecipient` to be required. * Added `new_name` field for `databricks.sdk.service.sharing.UpdateRecipient`. * Changed `name` field for `databricks.sdk.service.sharing.UpdateShare` to be required. * Added `new_name` field for `databricks.sdk.service.sharing.UpdateShare`. * Added `statement_ids` field for `databricks.sdk.service.sql.QueryFilter`. * Added `databricks.sdk.service.sql.StatementId` dataclass. OpenAPI SHA: 63caa3cb0c05045e81d3dcf2451fa990d8670f36, Date: 2023-12-12

strongbad03 · 2023-12-14T18:45:05Z

Ah, this explains why pages like the Groups page that I was using last week have since evaporated. I see its replacement, but I echo the concern with the remark above that losing examples is definitely a downside.

mgyucht added 5 commits November 30, 2023 18:46

First progress on major docs refactors

e42a2d1

more work

a2059ed

include inherited members in mixins

07156e2

examples

4e1c9c9

fix

de76431

mgyucht requested a review from pietern December 1, 2023 13:01

mgyucht commented Dec 1, 2023

View reviewed changes

pietern approved these changes Dec 1, 2023

View reviewed changes

mgyucht added 4 commits December 1, 2023 15:27

feedback

47ae8df

fix

2e0fcf3

more work

22932ac

last generation

589a3c0

mgyucht enabled auto-merge December 1, 2023 16:07

mgyucht added this pull request to the merge queue Dec 1, 2023

Merged via the queue into main with commit efe5ede Dec 1, 2023

mgyucht deleted the docs-improvements branch December 1, 2023 16:10

This was referenced Dec 1, 2023

[FEATURE] Documentation for Objects in the SDK #400

Closed

Install local library when generating docs #469

Merged

hectorcast-db mentioned this pull request Dec 12, 2023

Release v0.15.0 #478

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reference Documentation Refactoring#467

Reference Documentation Refactoring#467
mgyucht merged 9 commits intomainfrom
docs-improvements

mgyucht commented Dec 1, 2023

Uh oh!

mgyucht Dec 1, 2023

Uh oh!

mgyucht Dec 1, 2023

Uh oh!

mgyucht Dec 1, 2023

Uh oh!

pietern left a comment

Uh oh!

strongbad03 commented Dec 14, 2023 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

mgyucht commented Dec 1, 2023

Changes

Tests

Uh oh!

mgyucht Dec 1, 2023

Choose a reason for hiding this comment

Uh oh!

mgyucht Dec 1, 2023

Choose a reason for hiding this comment

Uh oh!

mgyucht Dec 1, 2023

Choose a reason for hiding this comment

Uh oh!

pietern left a comment

Choose a reason for hiding this comment

Uh oh!

strongbad03 commented Dec 14, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

strongbad03 commented Dec 14, 2023 •

edited

Loading